Key implementation areas of quality and risk management in a digital census

Risk and quality management is a continuous and integrated theme and should be woven into every stage of the census, ensuring that quality considerations guide all activities.

Planning and Design

Quality should be built into the census from the outset through:

Careful design of questionnaires and data collection instruments, adhering to international standards and best practices.
During pre-enumeration testing phase, conduct thorough testing of the questionnaire design, data collection software (CAPI application), and mapping systems including usability testing and error detection capabilities.
Establishment of clear data quality standards and procedures, aligned with the overall census objectives.

The potential risks during the planning and design stage may be categorised as:

Operational risks arising from delays in procurement, recruitment, or training. These risks could be due to challenges like pandemics, climatic conditions etc.
Technological risks arising from failures in digital tools, data loss, or cyberattacks.
Political and legal risks arising from changes in government priorities or legal challenges, and social risks arising from public mistrust, misinformation, or resistance to participation.

Risk mitigation strategies include:

Building public trust and using transparency to address concerns about data privacy and ensure public confidence in the census process.
Developing backup plans for critical systems and processes.
Ensuring clear mandates and data protection laws are in place.
Testing and piloting all digital tools and infrastructure adequately before full deployment.
Monitoring the entire census processes to ensure cost effectiveness and the quality of overall census operations crucial.

Training

Comprehensive training of census personnel is essential for data quality assurance by:

Ensuring enumerators and supervisors receive comprehensive training to promote consistent application of enumeration procedures and data definitions.
Minimizing errors in data collection and entry through hands-on practice and real-world scenarios.
Promoting understanding of data confidentiality and ethical considerations, emphasizing the importance of accurate and unbiased data collection.

The US Census Bureau emphasises that training is a high-risk phase in census operations due to its scale and time sensitivity. Proactive risk management ensures that training builds confidence and competence rather than confusion. Some training-related risks may include devices or software malfunction, inadequate internet access can disrupt virtual sessions, trainees may struggle with new tools or platforms and the risk of unauthorised access to training materials or personal data. To mitigate these risks:

Run small-scale trials (pilots) to test content delivery, tech platforms, and participant engagement.
Combine in-person and online training to accommodate varying tech access and skills.
Provide real-time technical support during training through help desks & support teams.
Train staff on secure login practices, password management, and data handling through clear cyber hygiene protocols.
For contingency planning, prepare printed manuals or offline training kits as backup devices and materials. Have alternative platforms or secondary tools ready in case the primary system fails.

For monitoring and evaluation during training, the following techniques can be adopted:

Use tests and quizzes to assess understanding and identify gaps and ensure feedback is given to the trainees.
Track attendance, completion rates, and technical issues in real time through performance dashboards.
Evaluate trainers’ effectiveness and adjust methods accordingly through trainer assessments.

Data Collection

Digital tools enable quality management by providing:

Real-time data validation to identify and correct errors during enumeration, reducing the need for post-enumeration editing. Validation rules can be used to check for:
- Completeness (ensuring all required fields are filled).
- Consistency (checking for logical inconsistencies between responses).
- Accuracy (verifying data against predefined ranges or lists).
- Examples of validation checks include:
  - Age within a reasonable range.
  - Consistency between age and education level.
Monitoring data collection progress and coverage using dashboards and geospatial tools in real-time.
Supervisory checks through digital platforms to monitor enumerator performance and identify potential issues. For example, supervisor field case management should encompass the methods by which interviewers receive assignments and manage questionnaire completeness. Field supervisors should oversee the completeness and quality of interviewer outputs, upload it for management and administrative review. The higher-level field managers should be able to track the progress of survey operations.

In terms of census monitoring, the use of independent monitors drawn from development partners, donors, Civil Society Organisations, National Statistical Offices (NSOs) in Africa and beyond, academia and other government agencies, to monitor enumeration should be considered as an additional quality assurance activity. Independent monitoring in representatively selected Enumeration Areas (EAs) can:

Objectively evaluate the census against international standards and national legislation and provide regular feedback to the NSO during and after the census activities.
Document the census process, lessons learnt and good practices for building capacity in future censuses.

Further information on independent census monitoring can be found in the Guidelines for Independent Census Monitoring in Africa 2030 Round of Population and Housing Censuses.

To manage risk during data collection, prepare backup systems and alternative data collection methods in case of technical failures as a contingency plan.

Data Processing

The quality of the data and thus outputs from the data is enhanced through:

Automated data cleaning and editing procedures to remove inconsistencies and errors.
Rigorous data validation and consistency checks using statistical software and algorithms.
Documentation of the processing steps for transparency and reproducibility, ensuring that data transformations are well-defined and auditable.

Risk management during data processing in digital census is crucial to ensure accuracy, security, and efficiency. Potential risks may include data corruption due to errors in transmission or storage leading to inaccurate results, cybersecurity threats from unauthorised access, hacking, or data breaches, system overloads or software failures, and automated processing errors from algorithm bias in the data editing script affecting data integrity. As a risk mitigation strategy:

Adopt cybersecurity measures to protect census data from cyber threats through encryption, access controls, and regular security audits.
Regularly audit data editing algorithms for fairness and accuracy.
Maintain backup servers and alternative processing methods or centres by establishing secondary locations for data handling. It can be in another government entity.
Plan for system failures with a documented disaster recovery plan.

For monitoring and evaluation during data processing, the following techniques can be adopted:

Use dashboards to monitor processing errors for real-time error tracking.
Maintain audit trails or logs of data modifications for accountability.
Conduct external/independent reviews and audits to verify data integrity.

Assessment of data quality

Post-enumeration, the quality of the enumeration can be assessed through various activities including:

Comparison of overall responses from the target population with population projections or administrative data.
Evaluating consistency and completeness at the question level.
Measuring the quality of any coding undertaken by field staff e.g., occupation coding
Identifying and resolving duplicate responses
Conducting a post-enumeration survey (PES) to gauge the level of under enumeration or over enumeration of people and dwellings.
Reviewing feedback from field staff.
Conducting an independent observation of the enumeration.

With the increasing shift to digital methodologies, the risk landscape for these surveys has evolved, introducing new challenges alongside traditional ones. Effective risk management throughout the lifecycle of a digital PES is paramount to ensuring the validity of its results and, consequently, the quality of the census data. A digital PES can be broken down into several key stages, each with its own set of potential risks and each addressed accordingly.

Data Dissemination

Quality management extends to how data is presented and used by:

Ensuring data accuracy and reliability through rigorous quality control throughout the process.
Providing clear metadata and documentation to facilitate proper interpretation and use of the data.
Promoting data literacy and appropriate use through training and user support.

Risk management in digital data dissemination is crucial to ensure data integrity, security, and accessibility. Potential risks include unauthorised access and data breaches, wrong information and misinterpretation of results, system downtime or data loss, and non-adherence to data protection laws. In addition to quality management aspects, validating data accuracy before dissemination, securing data against cyber threats through access controls, and ensuring users understand the data correctly through clear communication protocols.